Marie-Hélène Burle
December 12, 2023
I won’t introduce here the benefits of using a good version control system such as Git
While Git is a wonderful tool for text files versioning (code, writings in markup formats), it isn’t a tool to manage changes to datasets
Several open source tools—each with a different structure and functioning—extend Git capabilities to track data: Git LFS, git-annex, lakeFS, Dolt, DataLad
Reproducible research and collaboration on data science and machine learning projects involve more than datasets management:
Experiments and the models they produce also need to be tracked
*hp = hyperparameter
How did we get performance17 again? 🤯
Large files (datasets, models…) are kept outside Git
Each large file or directory put under DVC tracking has an associated .dvc file
Git only tracks the .dvc files (metadata)
Workflows can be tracked for collaboration and reproducibility
DVC functions as a Makefile and allows to only rerun what is necessary
pip:
conda
pipx (if you want dvc available everywhere without having to activate virtual envs):
Optional dependencies [s3], [gdrive], etc. for remote storage
Terminal
Python library if installed via pip or conda
In this webinar, I will use DVC through the command line
Code and data for this webinar modified from:
├── LICENSE
├── data
│ ├── prepared
│ └── raw
│ ├── train
│ └── val
├── metrics
├── model
├── requirements.txt
└── src
├── evaluate.py
├── prepare.py
└── train.py
Initialized empty Git repository in dvc/.git/
This creates the .git directory
Initialized DVC repository.
You can now commit the changes to git.
You will also see a note about usage analytics collection and info on how to opt out
A .dvc directory and a .dvcignore file got created
DVC automatically staged its system file for us:
On branch main
No commits yet
Changes to be committed:
new file: .dvc/.gitignore
new file: .dvc/config
new file: .dvcignore
Untracked files:
LICENSE
data/
requirements.txt
src/
So we can directly commit:
Let’s work in a virtual environment:
Let’s clean things up:
git add .gitignore LICENSE requirements.txt
git commit -m "Add general files"
git add src
git commit -m "Add scripts"On branch main
Untracked files:
data/
Now, it is time to deal with the data
We are still not tracking any data:
There are no data or pipelines tracked in this project yet.
You can choose what to track as a unit (i.e. each picture individually, the whole data directory as a unit)
Let’s break it down by set:
This added data to .dvc/cache and created 3 files in data/raw:
.gitignoretrain.dvcval.dvcThe .gitignore tells Git not to track the data:
/train
/val
The .dvc files contain the metadata for the cached directories
We are all good:
Data and pipelines are up to date.
Link between checked-out version of a file/directory and the cache:
| Duplication | Editable | |
|---|---|---|
| Reflinks* | Only when needed | Yes |
| Hardlinks/Symlinks | No | No |
| Copies | Yes | Yes |
*Reflinks only available for a few file systems (Btrfs, XFS, OCFS2, or APFS)
The metafiles should be put under Git version control
You can configure DVC to automatically stage its newly created system files:
You can then commit directly:
On branch main
nothing to commit, working tree clean
Let’s make some change to the data:
Remember that Git is not tracking the data:
Data and pipelines are up to date.
What if we want to go back to the 1st version of our data?
For this, we first use Git to checkout the proper commit, then run dvc checkout to have the data catch up to the .dvc file
To avoid forgetting to run the commands that will make DVC catch up to Git, we can automate this process by installing Git hooks:
Now, all we have to do is to checkout the commit we want:
94b520b (HEAD -> main) Delete data/raw/val/n03445777/ILSVRC2012_val*
92837a6 Initial version of data
dd961c6 Add scripts
db9c14e Initialize repo
7e08586 Initialize DVC
The version of the data in the working directory got automatically switched to match the .dvc file:
Data and pipelines are up to date.
You can look at your files to verify that the deleted files are back
git checkout is ok to have a look, but a detached HEAD is not a good place to create new commits
Let’s create a new branch and switch to it:
Switched to a new branch 'alternative'
The Git project (including .dvc files) go to a Git remote (GitHub/GitLab/Bitbucket/server)
The data go to a DVC remote (AWS/Azure/Google Drive/server/etc.)
DVC can use many cloud storage or remote machines/server via SSH, WebDAV, etc.
Let’s create a local remote here:
# Create a directory outside the project
mkdir ../remote
# Setup default (-d) remote
dvc remote add -d local_remote ../remoteSetting 'local_remote' as a default remote.
[core]
remote = local_remote
['remote "local_remote"']
url = ../../remote
The new remote configuration should be committed:
On branch alternative
Changes not staged for commit:
modified: .dvc/config
Let’s push the data from the cache (.dvc/cache) to the remote:
2702 files pushed
With Git hooks installed, dvc push is automatically run after git push
(But the data is pushed to the DVC remote while the files tracked by Git get pushed to the Git remote)
By default, the entire data cache gets pushed to the remote, but there are many options
dvc fetch downloads data from the remote into the cache. To have it update the working directory, follow by dvc checkout
You can do these 2 commands at the same time with dvc pull
DVC pipelines create reproducible workflows and are functionally similar to Makefiles
Each step in a pipeline is created with dvc stage add and add an entry to a dvc.yaml file
This entry contains:
cmd: the command executeddeps: the dependenciesouts: the outputsThe file is then used to visualize the pipeline and run it
Let’s create a pipeline to run a classifier on our data
The pipeline contains 3 steps:
dvc stage add -n prepare \
-d src/prepare.py -d data/raw \
-o data/prepared/train.csv -o data/prepared/test.csv \
python src/prepare.pyAdded stage 'prepare' in 'dvc.yaml'
dvc stage add -n train \
-d src/train.py -d data/prepared/train.csv \
-o model/model.joblib \
python src/train.pyAdded stage `train` in 'dvc.yaml'
dvc stage add -n evaluate \
-d src/evaluate.py -d model/model.joblib \
-M metrics/accuracy.json \
python src/evaluate.pyAdded stage `evaluate` in 'dvc.yaml'
+---------+
| prepare |
+---------+
*
*
*
+-------+
| train |
+-------+
*
*
*
+----------+
| evaluate |
+----------+
DVC is a sophisticated tool with many additional features:
Creation of data registries
A Python library to log experiment metrics
Visualize the performance logs as plots
Continuous integration
With the sister project CML (Continuous Machine Learning)